Automated assessment of aircraft structure damage

ABSTRACT

A system may include a camera module that may include a first infrared camera, a second infrared camera, and a visible light camera. The system may further include an autonomous vehicle that may include a vertically extendable arm attached to the camera module. The system may also include a processor configured to initiate movement of the autonomous vehicle around an aircraft according to a predetermined path, initiate a scan of an exterior surface of the aircraft using the first infrared camera, the second infrared camera, the visible light camera, or a combination thereof, determine whether a portion of the exterior surface of the aircraft is damaged based on the scan, and in response to the portion of the exterior surface of the aircraft being damaged, use the first infrared camera, the second infrared camera, and the visible light camera to generate a three-dimensional model of the portion of the exterior surface of the aircraft.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application, under 35 U.S.C. § 119, claims the benefit of U.S.Provisional Patent Application Ser. No. 63/165,387 filed on Mar. 24,2021, and entitled “Automated Assessment of Aircraft Structure Damage,”the contents of which are hereby incorporated by reference herein.

FIELD OF THE DISCLOSURE

This disclosure is generally related to the field of aircraft inspectionand, in particular, to automated assessment of aircraft structuredamage.

BACKGROUND

Inspection and maintenance may keep an airplane and its components underoptimal working conditions, thereby upholding safety standards set byaviation regulatory authorities. As part of an inspection andmaintenance routine, an entire aircraft may be examined, maintained, andhave parts replaced or repaired. Damage assessment may be part of theflight decision-making processes to ensure flight safety and may beperformed during field services, repairs, and ramp inspections of anaircraft, whether it is at a maintenance hangar or parked at an airportterminal gate. Airline ground crews and the flight crew are typicallyresponsible to ensure the aircraft is airworthy before pushback fordepartures.

Crewmembers typically conduct a ground inspection of parked aircraft atan airport terminal gate. The inspection, known as airport rampinspection, may involve checks on the aircraft's structure and systemcomponents for visible damages. A walk-around or pre-flight visualinspection may be manually conducted during a quick-turnaround at theairport gate. The top of the fuselage, the top of the wings, and the topof the stabilizers may not be inspected until the airplane is placed inlayovers. When airport visibility or light conditions are poor,inspection items may be difficult to discern. Further, in order toassess foreign object damage (FOD) caused by hail, birds, lightning,runway debris, etc., resulting in structural damages, a crewmembertypically climbs on a ladder and uses measurement tools and gauges tomanually obtain dimensions such as width and depth of the damaged areas.The measurements may be manually recorded, along with the crewmember'sconclusions, before a decision for departure can be made. This addedtime and effort may contribute to flight delays, also potential crewinjuries. Other disadvantages may exist.

SUMMARY

Disclosed are systems and methods for automated assessment of aircraftstructure damage. In an example, a system includes a camera moduleincluding a first infrared camera, a second infrared camera, and avisible light camera. The system further includes an autonomous vehicleincluding a vertically extendable arm, where the camera module isattached to a distal end of the vertically extendable arm. The systemalso includes a processor configured to initiate movement of theautonomous vehicle around an aircraft according to a predetermined path.The processor is also configured to initiate a scan of an exteriorsurface of the aircraft using the first infrared camera, the secondinfrared camera, the visible light camera, or a combination thereof. Theprocessor is further configured to determine whether a portion of theexterior surface of the aircraft is damaged based on the scan, and inresponse to the portion of the exterior surface of the aircraft beingdamaged, use the first infrared camera, the second infrared camera, andthe visible light camera to compute dimensional parameters of damage tothe exterior surface and generate a three-dimensional model of theportion of the exterior surface of the aircraft.

In some examples, the camera module further includes a motorized tableconfigured to rotate in both azimuth and elevation, where the firstinfrared camera, the second infrared camera, and the visible lightcamera are attached to the motorized table, and where the first infraredcamera and the second infrared camera are separated by a fixed distancerelative to one another. In some embodiments, initiating the scan of theexterior surface of the aircraft includes commanding the camera moduleto scan azimuthally at incremental changes of elevation angles. In someexamples, the processor is further configured to plan the movement ofthe autonomous vehicle around the aircraft using a stored electronicmap, and verify the movement of the autonomous vehicle using detectablereferences positioned on a tarmac, a radar, a lidar, a globalpositioning system, or a combination thereof. In some examples, theautonomous vehicle is configured to use proximity range sensors to avoidobstacles during the movement of the autonomous vehicle around theaircraft. In some examples, the first infrared camera and the secondinfrared camera are each capable of infrared illumination usingmodulated continuous waves, where a modulated continuous wave associatedwith the first infrared camera is phase shifted to prevent interferencewith a modulated continuous wave of the second infrared camera, wherethe first infrared camera and the second infrared camera are eachcapable of taking independent time-of-flight measurements using themodulated continuous waves, and where the independent time-of-flightmeasurements are used in generating the three-dimensional model of theportion of the exterior surface. In some examples, the processor isfurther configured to perform a differential measurement process usingvisible image data from the visible light camera to remove an ambientbackground light component from infrared image data received from thefirst infrared camera and the second infrared camera.

To illustrate the principle of automated assessment of aircraft surfacedamage, several methods of assessing structural damages such as dents onaircraft exterior are explained throughout this disclosure. In someexamples, in response to the portion of the exterior surface of theaircraft being damaged, the processor is further configured to initiatemovement of the autonomous vehicle to position the first infrared cameraand the second infrared camera so that a base line between the firstinfrared camera and the second infrared camera is parallel with theportion of the exterior surface of the aircraft. The processor may beconfigured to rotate the first infrared camera and the second infraredcamera so that the first infrared camera and the second infrared cameraare iteratively directed at points running along a dent in the portionof the exterior surface of the aircraft beginning at a starting point ona first side of the dent and ending at an ending point on a second sideof the dent. For each point of the points running along the dent, theprocessor may determine a first angle in azimuth associated with thefirst infrared camera and a second angle in azimuth associated with thesecond infrared camera. The processor may calculate distances betweenthe base line and the points running along the dent using the firstangle in azimuth and the second angle in azimuth for each point of thepoints, where the distances between the base line and the points runningalong the dent are used to generate the three-dimensional model of theportion of the exterior surface of the aircraft.

In some examples, in response to the portion of the exterior surface ofthe aircraft being damaged, the processor is further configured todetermine a first distance between the first infrared camera and astarting point of a dent in the portion of the exterior surface of theaircraft using a first time-of-flight measurement. The processor mayalso determine a second distance between the first infrared camera andan ending point of the dent using a second time-of-flight measurement.The processor may determine an angle in azimuth between a firstdirection associated with the first infrared camera being directed atthe starting point of the dent and a second direction associated withthe first infrared camera being directed at the ending point. Theprocessor may further calculate a width between the starting point ofthe dent and the ending point of the dent based on the first distance,the second distance, and the angle in azimuth. In response to theportion of the exterior surface of the aircraft being damaged, theprocessor may be configured to use the second infrared camera to confirmthe width between the starting point of the dent and the ending point ofthe dent

In response to the portion of the exterior surface of the aircraft beingdamaged, the processor may be configured to initiate movement of theautonomous vehicle to position the first infrared camera and the secondinfrared camera so that a base line between the first infrared cameraand the second infrared camera is parallel with the portion of theexterior surface of the aircraft. The processor may further determine afirst distance between the first infrared camera and a starting point ofa dent in the portion of the exterior surface of the aircraft using afirst time-of-flight measurement. The processor may also determine asecond distance between the first infrared camera and an ending point ofthe dent using a second time-of-flight measurement. The processor maydetermine a third distance between the second infrared camera and thestarting point of the dent using a third time-of-flight measurement. Theprocessor may further determine a fourth distance between the secondinfrared camera and the ending point of the dent using a fourthtime-of-flight measurement. The processor may calculate a width betweenthe starting point of the dent and the ending point of the dent based onthe first distance, the second distance, the third distance, the fourthdistance, and a fifth distance between the first infrared camera and thesecond infrared camera. In some examples, the processor is furtherconfigured to produce an augmented reality image or video thatsuperimposes dimensional parameters from the three-dimensional modelonto an image or video of the portion of the exterior surface of theaircraft using the three-dimensional model. In some examples, theprocessor is further configured to enhance visible image data generatedby the visible light camera by using a grayscale conversion to removediscrepancies due to lighting conditions.

In some examples, the processor is further configured to provide liveimagery or video from the visible light camera to a mobile device via anetwork connection. In some examples, the processor is furtherconfigured to provide the three-dimensional model to a mobile device forrendering via a network connection. In some examples, the systemincludes a database, where the processor is further configured toprovide the three-dimensional model to the database for use incomparisons to future three-dimensional models associated with theportion of the exterior surface, to provide findings related to damageto the exterior surface to the database for documentation, or both. Insome embodiments, the processor is configured to provide a damageassessment to a mobile device to assist a crew, an airline, or both inmaking flight decisions.

In an example, a method includes moving an autonomous vehicle around anaircraft according to a predetermined path, where the autonomous vehiclecomprises a vertically extendable arm, where a camera module is attachedto a distal end of the vertically extendable arm, and where the cameramodule comprises a first infrared camera, a second infrared camera, anda visible light camera. The method further includes scanning an exteriorsurface of the aircraft using the first infrared camera, the secondinfrared camera, the visible light camera, or a combination thereof. Themethod also includes determining whether a portion of the exteriorsurface of the aircraft is damaged based on the scan. The methodincludes, in response to the portion of the exterior surface of theaircraft being damaged, using the first infrared camera, the secondinfrared camera, and the visible light camera to generate athree-dimensional model of the portion of the exterior surface of theaircraft.

In some examples, the method may include initiating movement of theautonomous vehicle to position the first infrared camera and the secondinfrared camera so that a base line between the first infrared cameraand the second infrared camera is parallel with the portion of theexterior surface of the aircraft. The method may further includerotating the first infrared camera and the second infrared camera sothat the first infrared camera and the second infrared camera areiteratively directed at points running along a dent in the portion ofthe exterior surface of the aircraft beginning at a starting point on afirst side of the dent and ending at an ending point on a second side ofthe dent. The method may include, for each point of the points runningalong the dent, determining a first angle in azimuth associated with thefirst infrared camera and a second angle in azimuth associated with thesecond infrared camera. The method may include calculating distancesbetween the base line and the points running along the dent using thefirst angle in azimuth and the second angle in azimuth for each point ofthe points, wherein the distances between the base line and the pointsrunning along the dent are used to generate the three-dimensional modelof the portion of the exterior surface of the aircraft.

In some examples, the method includes determining a first distancebetween the first infrared camera and a starting point of a dent in theportion of the exterior surface of the aircraft using a firsttime-of-flight measurement. The method may further include determining asecond distance between the first infrared camera and an ending point ofthe dent using a second time-of-flight measurement. The method mayinclude calculating a width between the starting point of the dent andthe ending point of the dent based, at least in part, on the firstdistance and the second distance.

In an example, an apparatus includes a camera module including a firstinfrared camera, a second infrared camera, and a visible light camera.The apparatus includes an autonomous vehicle including a verticallyextendable arm, where the camera module is attached to a distal end ofthe vertically extendable arm, where the autonomous vehicle isconfigured to move around an aircraft according to a predetermined path,and where the camera module is configured to scan of an exterior surfaceof the aircraft using the first infrared camera, the second infraredcamera, the visible light camera, or a combination thereof.

In some examples, the autonomous vehicle is further configured to use astored electronic map, detectable references positioned on a tarmac, aradar, a lidar, a global positioning system, or a combination thereofwhile moving, and where the autonomous vehicle is further configured touse proximity range sensors to avoid obstacles during movement of theautonomous vehicle around the aircraft.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts an example of a system for automated assessment ofaircraft structure damage in conjunction with an aircraft.

FIG. 2 depicts an example of a predetermined path around an aircraft.

FIG. 3 depicts an example of an apparatus for automated assessment ofaircraft structure damage.

FIG. 4 depicts an example of a motorized table for an example of asystem for automated assessment of aircraft structure damage.

FIG. 5 depicts an example of a system for automated assessment ofaircraft structure damage.

FIG. 6 depicts an example of a first set of measurements taken over anexterior surface for determining a depth of a dent.

FIG. 7 depicts an example of a second set of measurements taken over anexterior surface determining a depth of a dent.

FIG. 8 depicts an example of a first set of measurements taken over anexterior surface for determining a width of a dent.

FIG. 9 depicts an example of a second set of measurements taken over anexterior surface for determining a width of a dent.

FIG. 10 depicts an example of an augmented reality image of a surface.

FIG. 11 is an operation flow chart for automated aircraft inspectionduring gate turnaround at airport.

FIG. 12 is an inspection flow chart depicting an example of a method forautomated assessment of aircraft structure damage.

While the disclosure is susceptible to various modifications andalternative forms, specific examples have been shown by way of examplein the drawings and will be described in detail herein. However, itshould be understood that the disclosure is not intended to be limitedto the particular forms disclosed. Rather, the intention is to cover allmodifications, equivalents and alternatives falling within the scope ofthe disclosure.

DETAILED DESCRIPTION

Referring to FIG. 1, an example of a system 100 for automated assessmentof aircraft structure damage is depicted. The system 100 may be used toassess an exterior surface 132 of an aircraft 130. The exterior surface132 of the aircraft 130 may include any externally visible portion ofthe aircraft that is capable of being assessed through imagery.

The system 100 may include an apparatus 102 which may include a cameramodule 110 and an autonomous vehicle 120. The camera module 110 may beconfigured to capture both visible light and infrared imagery and/orvideos. The autonomous vehicle 120 may include a vertically extendablearm 122, which may attach to the autonomous vehicle 120 at a proximalend 124 of the vertically extendable arm 122, and which may attach tothe camera module 110 at a distal end 126 of the vertically extendablearm 122.

During operation, the autonomous vehicle 120 may be used to position thecamera module 110 to enable inspection of the aircraft 130. For example,the autonomous vehicle 120 may move to a position near the aircraft 130and the vertically extendable arm 122 may lift the camera module 110 toa height sufficient to view the exterior surface 132 of the aircraft130.

A benefit of the system 100 is that ground inspections can be performedboth above and below the aircraft 130. Further, since drone assistedaircraft inspections are generally prohibited at civil airports,performing a comprehensive surface inspection of the upper part ofaircraft 130 may be difficult. However, the vertically extendable arm122 may overcome this difficulty by enabling aerial views of theaircraft 130 similar to those that a drone may provide. Other benefitsmay exist as described herein.

Referring to FIG. 2, an example of a predetermined path 202 is depictedaround an aircraft 130. The autonomous vehicle 120, from FIG. 1, may beconfigured to move around the aircraft 130 according to thepredetermined path 202. Detectable references 204 positioned on a tarmacmay assist the autonomous vehicle 120 in staying on the predeterminedpath 202. For example, the autonomous vehicle 120 may include anothercamera and/or sensor to detect the detectable references 204. In somecases, the camera module 110 may be used to detect the detectablereferences 204. The detectable references 204 may include dots, strips,reference waypoints, or combinations thereof placed on the tarmac. As analternative to, or in addition to, the detectable references 204, theautonomous vehicle 120 may be configured to use a stored electronic map,radar, lidar, a global positioning system, or a combination thereof toguide it along the predetermined path 202.

While moving, the autonomous vehicle 120 may be configured to useproximity range sensors to avoid obstacles 206. Examples of obstaclesthat the autonomous vehicle 120 may encounter include, but are notlimited to, ground crew members, ground vehicles, and portions of theaircraft 130. Upon encountering the obstacles 206, the autonomousvehicle 120 may be configured to stop and wait until the obstacles 206are removed or the autonomous vehicle 120 may be configured to plan andtake detour paths 208 to move around the obstacles 206. After followingthe detour paths 208, the autonomous vehicle 120 may continue along thepredetermined path 202.

During the movement along the predetermined path 202, the camera module110 may be configured to scan the exterior surface 132 of the aircraft130. The scan may be used, as further described herein, to assess damageto the exterior surface 132, thereby providing a ground crew and aflight crew with sufficient data to make aircraft service and flightdecisions.

Referring to FIG. 3, an example of an apparatus 102 for automatedassessment of aircraft structure damage is depicted. The apparatus 102may include a camera module 110, an automated vehicle 120, and avertically extendable arm 122. The vertically extendable arm 122 mayattach to the automated vehicle 120 at a proximal end 124 of thevertically extendable arm 122 and may attach to the camera module 110 ata distal end 126 of the vertically extendable arm 122. Thus, thevertically extendable arm 122 may raise and lower the camera module 110as desired.

The camera module 110 may include a first infrared camera 302, a secondinfrared camera 304, and a visible light camera 306. The first infraredcamera 302 and the second infrared camera 304 may each be capable ofinfrared illumination using modulated continuous waves. Further, themodulated continuous waves associated with the first infrared camera 302may be phase shifted to prevent interference with the modulatedcontinuous waves associated with the second infrared camera 304. Thismay enable the first infrared camera and the second infrared camera tocapture image data independent of each other. Further, the firstinfrared camera 302 and the second infrared camera 304 may each becapable of taking independent time-of-flight measurements using themodulated continuous waves having different phase shifts. Thesetime-of-flight measurements may be used in generating athree-dimensional model of damaged portions of an aircraft as describedfurther herein.

The camera module 110 may further include a motorized table 310. Thefirst infrared camera 302, the second infrared camera 304, and thevisible light camera 306 may be attached to, or otherwise mounted on,the motorized table 310, such that they may be rotated in both azimuthand elevation by the motorized table 310. The first infrared camera 302and the second infrared camera 304 may be separated by a fixed distance308 relative to one another to enable various measurements to beperformed as described herein.

Referring to FIG. 4, an example of a motorized table 310 is depicted.The motorized table 310 may include a first rotatable portion 402 toenable rotation in azimuth Ø and a second rotatable portion 404 toenable rotation in elevation θ. The motorized table 310 may enable thecamera module 110 of FIGS. 1 and 3 to be directed at the exteriorsurface 132 from multiple directions. Changes of angles in both azimuthØ and elevation θ of FIG. 4 enables the camera module 110 to scan theexterior surface 132, or any particular area of interest.

Referring to FIG. 5, an example of a system 100 for automated assessmentof aircraft structure damage is depicted. As described herein, thesystem 100 may include a camera module 110 and an autonomous vehicle120. The camera module 110 may include a first infrared camera 302, asecond infrared camera 304, a visible light camera 306, and a motorizedtable 310. The first infrared camera 302 and the second infrared camera304 may each be capable of infrared illumination using modulatedcontinuous waves. For example, a first modulated continuous wave 512associated with the first infrared camera 302 may be phase shifted toprevent interference with a second modulated continuous wave 514associated with the second infrared camera 304. In that way, the firstinfrared camera 302 and the second infrared camera 304 may each becapable of taking time-of-flight measurements 550 that are independentof each other.

The autonomous vehicle 120 may include a vertically extendable arm 122coupled to the camera module 110. The vehicle 120 may further include aradar 522, a lidar 524, a global positioning system 526, and/orcombinations thereof to assist the vehicle 120 in following thepredetermined path 202 of FIG. 2. The autonomous vehicle 120 may alsoinclude proximity range sensors 528 to assist the vehicle 120 withavoiding the obstacles 206 of FIG. 2.

The system 100 may include a control module 530 for controlling theautonomous vehicle 120 and for processing data as described herein. Thecontrol module 530 may be part of the autonomous vehicle 120, or it maybe separate from the autonomous vehicle 120 and may perform controlfunctions remotely. In some cases, the control module 530 may bedistributed between the autonomous vehicle 120 and one or more remotedevices.

The control module 530 may include a processor 532. As used herein, theprocessor 532 may include a microcontroller, a central processing unit(CPU), a graphical processing unit (GPU), a digital signal processor(DSP), a peripheral interface controller (PIC), another type ofmicroprocessor, and/or combinations thereof. Further, the processor 532may be implemented as an integrated circuit, a complementarymetal-oxide-semiconductor (CMOS) metal-oxide-semiconductorfield-effect-transistor (MOSFET) circuit, a very-large-scale-integrated(VLSI) circuit, a field-programmable gate array (FPGA), anapplication-specific integrated circuit (ASIC), a combination of logicgate circuitry, another type of digital or analog electrical designcomponent, or combinations thereof. For purposes of this disclosure, theprocessor 532 may further comprise memory sufficient to perform thefunctions described herein. The memory may include memory devices suchas random-access memory (RAM), read-only memory (ROM), magnetic diskmemory, optical disk memory, flash memory, another type of memorycapable of storing data and processor instructions, or the like, orcombinations thereof.

The system 100 may include a mobile device 562 and a database 564coupled to the control module 530 via a network connection 560. Themobile device 562 may be, but is not limited to, a laptop device, atablet device, an electronic flight book, or another type of personalcomputing device. The mobile device 562 may be associated with a groundcrew member, a flight crew member, or another member of a group chargedwith assessing an aircraft.

During operation, the processor 532 may initiate movement of theautonomous vehicle 120 around an aircraft (e.g., the aircraft 130)according to a predetermined path 202. Before initiating the movement,the processor 532 may plan the movement of the autonomous vehicle 120using a stored electronic map 534. As the autonomous vehicle 120 moves,the processor 532 may verify the movement of the autonomous vehicle 120using detectable references (e.g., the detectable references 204)positioned on a tarmac. The movement may also be verified through theuse of the radar 522, the lidar 524, the global positioning system 526,or a combination thereof. Further, the processor 532 may receive inputfrom the proximity range sensors 528 to avoid obstacles (e.g., theobstacles 206) during the movement of the autonomous vehicle 120 aroundthe aircraft 130.

While the autonomous vehicle 120 moves around the aircraft 130, theprocessor 532 may initiate a scan 536 of an exterior surface 132 of theaircraft 130 using the first infrared camera 302, the second infraredcamera 304, the visible light camera 306, or a combination thereof. Insome cases, the scan 536 may be made using the visible light camera 306and the processor 532 may rely on visual image processing techniques toanalyze the scan 536. In this way, the processor 532 may determinewhether a portion of the exterior surface 132 of the aircraft 130 isdamaged.

In response to determining that damage exists, the control module 530may receive the time-of-flight measurements 550, visible image data 552,and infrared image data 556 from the camera module 110. The processor532 may use the time-of-flight measurements 550 to generate athree-dimensional model 542 of the damage. The processor 532 may alsogenerate an augmented reality image or video 544 based on thethree-dimensional model 542, the visible image data 552, the infraredimage data 556, or combinations thereof. The augmented reality image orvideo 544 may be used to aid crews in making flight decisions.

Various processes may be applied to the visible image data 552 and theinfrared image data 556 to improve its usefulness for flight and groundcrews. For example, the visible image data 552 may initially includediscrepancies due to lighting conditions 554. The processor 532 mayenhance the visible image data 552 generated by the visible light camera306 by using a grayscale conversion 540 to remove the discrepancies dueto lighting conditions 554. As another example, the infrared image data556 may include an ambient background light component 558. The processor532 may perform a differential measurement process 538 using the visibleimage data 552 from the visible light camera 306 to remove the ambientbackground light component 558. The differential measurement process 538may differentiate the visible image data 552 taken by the visible lightcamera 306 and the infrared image data 556 and reveal infrared-onlycontent. The resulting content is without the effects of the ambientbackground light component 558 which may contribute to errors such asbias, shift, and offset. Based on measured pixel intensity, phase anglesbetween illumination and reflection and a distance between the imagingsensor and an object can be calculated.

The control module 530 may be configured to provide a live image orvideo 545 from the visible light camera 306 to a mobile device 562 via anetwork connection 560. This may enable a flight crew or a ground crewto view the scan 536 in real time. Further, the control module 530 maybe configured to provide the three-dimensional model 542 to the mobiledevice 562 for rendering via the network connection 560. Thethree-dimensional model 542 may be superimposed with the augmentedreality image or video 544. The control module 530 may provide thethree-dimensional model 542 to the database 564. The database 564 maystore previous three-dimensional models 568 to use for comparison. Thus,the three-dimensional model 542 may be stored for use in comparisons tofuture three-dimensional models. The control module 530 may furtherprovide a damage assessment to the mobile device 562 to assist a crew,an airline, or both in making flight decisions.

A benefit of the system 100 is that aircraft inspections may beautomated and efficiency of flight crews and ground crews in makingdamage assessments can be increased. Additionally, by maintaining thedatabase 564, more data for analytics may be generated. Other benefitsmay exist.

Referring to FIG. 6, an example of a first set of measurements takenover an exterior surface 132 for determining a depth of a dent 602 isdepicted. The measurements depicted may be taken in response to adetermination that a portion 601 of the exterior surface 132 (e.g., ofthe aircraft 130) is damaged.

In order to determine the depth and/or shape of the dent 602, a baseline 610 between a first point 622 (which may be associated with thefirst infrared camera 302 of FIGS. 3 and 5) and a second point 624(which may be associated with the second infrared camera 304 of FIGS. 3and 5) may be made parallel with the portion 601 of the exterior surface132. In order to position the base line 610 in a parallel configuration,the autonomous vehicle 120 of FIGS. 1, 3, and 5 may be moved. Inparticular, the processor 532 may initiate movement of the autonomousvehicle 120.

The first infrared camera 302 and the second infrared camera 304 cameramay be rotated so that the first infrared camera 302 and the secondinfrared camera 304 are directed to a starting point 604 of the dent602. As shown in FIG. 6, a first angle of azimuth 614 may be determinedat the first point 622 based on an angle of rotation of the firstinfrared camera 302. A second angle of azimuth 618 may be determined atthe second point 624 based on an angle of rotation of the secondinfrared camera 304. A distance 626 between the first point 622 and thesecond point 624 may be fixed. A distance 620 between the base line 610and the starting point 604 may be determined based on the followingformula:

$h = \frac{d}{\left( {\tan\alpha} \right)^{- 1} + \left( {\tan\beta} \right)^{- 1}}$

where h is the distance 620 between the base line 610 and the point 604,d is the distance 626 between the first point 622 and the second point624 (e.g., between the first infrared camera 302 and the second infraredcamera 304), α is the first angle in azimuth 614, and β is the secondangle in azimuth 618. Since the distance 626 between the two infraredcameras is known, the process may be iteratively repeated for multiplepoints, such as at least one intermediate point 606, up to an endingpoint 608. The process may be repeated along a full plane associatedwith the exterior surface 132 to map out depths and shape of the dent602.

Referring to FIG. 7, an example of a second set of measurements takenover the portion 601 of the exterior surface 132 for determining a depthof the dent 602 is depicted. In FIG. 7, the first infrared camera 302and the second infrared camera 304 may be rotated so that the firstinfrared camera 302 and the second infrared camera 304 are directed tothe intermediate point 606. In this state, the first angle in azimuth614 at the first point 622 has been reduced relative to FIG. 6 and thesecond angle in azimuth 618 at the second point 624 has been increasedrelative to FIG. 6. Because the intermediate point 606 is deeper withinthe portion 601 of the exterior surface 132, a distance 620 between thebase line 610 and the intermediate point 606 may be increased relativeto FIG. 6. The distance 620 may be determined as described withreference to FIG. 6 based on the first angle in azimuth 614, the secondangle in azimuth 618, and the fixed distance 626 between the first point622 and the second point 624.

As described with respect to FIGS. 6 and 7, for each point (e.g., thestarting point 604, the intermediate point 606, and the ending point608) of a set of points running along the dent 602, a first angle inazimuth 614 associated with the first infrared camera 302 and a secondangle in azimuth 618 associated with the second infrared camera 304 maybe determined. Distances between the base line 610 and the pointsrunning along the dent 602 may be calculated using the first angle inazimuth 614 and the second angle in azimuth 618 for each point. Thisprocess may be repeated for incremental changes of elevation angle θ ofFIG. 4 to scan the entire area of the dent 606. Distances (e.g., thedistance 620) between the base line 610 and the points may be used togenerate the three-dimensional model 542 mapping out the depth and shapeof the dent 602.

Referring to FIG. 8, an example of a first set of measurements takenover a portion 601 of an exterior surface 132 for determining a width840 of a dent 602 is depicted. The width 840 can be determined usingmultiple different measurements, one of which is depicted in FIG. 8 andanother of which is depicted in FIG. 9.

In FIG. 8, a first time-of-flight measurement may be used to determine afirst distance 812 between the first point 622 associated with the firstinfrared camera 302 and a starting point 802 of the dent 602. A seconddistance 828 between the first point 622 and an ending point 804 of thedent 602 may be determined using a second time-of-flight measurement. Anangle in azimuth 814 between a first direction 813 associated with thefirst infrared camera 302 being directed at the starting point 802 ofthe dent 602 and a second direction 829 associated with the firstinfrared camera 302 being directed at the ending point 804 may bedetermined. The width 840 may be trigonometrically calculated throughthe law of cosines based on the first distance 812, the second distance828, and the angle of azimuth 814. In other words,

w=√{square root over (a ² +b ²−2×a×b×cos γ)}

where w is the width 840 between the starting point 802 and the endingpoint 804, a is the distance 812 between the first point 622 and thesecond point 802; b is the distance 828 between the first point 622 andthe second point 804, and γ is the angle of azimuth 814.

The second infrared camera 304 associated with the second point 624 maybe used to confirm the width 840. For example, the width 840 may betrigonometrically verified based on a third distance 830 between thesecond point 624 and the ending point 802, a fourth distance 816 betweenthe second point 624 and the starting point 804, and another angle inazimuth 818 associated with the second point 624.

Referring to FIG. 9, an example of a second set of measurements takenover the portion 601 of the exterior surface 132 for determining thewidth 840 of the dent 602 is depicted. In the case of FIG. 9, theprocess may be performed after movement of the autonomous vehicle 120 toposition the first infrared camera 302 and the second infrared camera304 so that a base line 610 between the first infrared camera 302 andthe second infrared camera 304 is parallel with the portion 601 of theexterior surface 132.

In FIG. 9, a first distance 812 between the first point 622 associatedwith the first infrared camera 302 and a starting point 802 of the dent602 may be determined using a first time-of-flight measurement. A seconddistance 928 between the first point 622 and an ending point 804 of thedent 602 may be determined using a second time-of-flight measurement. Athird distance 930 between the second point 624 and the starting point802 may be determined using a third time-of-flight measurement. A fourthdistance 816 between the second point 624 and the ending point 804 maybe determined using a fourth time-of-flight measurement. Under thiscondition that the camera module 110 is in parallel with the portion601, a quadrilateral based on the distances 812, 816, 928, 930 can betreated as a cyclic type. Based on the principle of Ptolemy Inequalityof a quadrilateral, the width 840 of the dent 602 may be determined inan alternative method that is based solely on distance measurements,without azimuth angle involved.

$w = \frac{{c \times g} - {a \times e}}{f}$

where w is the same width 840 between the starting point 802 and theending point 804; a is the same distance 812 between the first point 622and the starting point 802; c is the distance 928 between the firstpoint 622 and the ending point 804; g is the distance 930 between thesecond point 624 and the starting point 802; e is the distance 816between the second point 624 and the ending point 804; f is the distance626 between the first point 622 and the second point 624, i.e., theseparation between two infrared cameras on camera module 110.

By scanning the camera module 110 azimuthally at every incrementalchange of the elevation angles in FIG. 4, depths and widths of the dent602 are determined, and a three-dimensional model 542 of FIG. 5 may begenerated. Further, the augmented reality image or video 544 may begenerated based on the three-dimensional model 542.

Referring to FIG. 10, an example of an augmented reality image or video1000 is depicted. The augmented image or video 1000 may include avisible image or video 1002 superimposed with a mesh or a grid 1004representing damage information. For example, as depicted in FIG. 10,depth numbers 1006 may be present along the grid 1004 to indicate thecontour and/or shape of damage depicted in the image or video 1002. Theaugmented reality image or video 1000 may enable a flight crew member ora ground crew member to make an accurate assessment of the effect thedamage may have on a flight. Additional flight decisions may then bemade accordingly.

Referring to FIG. 11, an example of a method 1100 for automatedassessment of aircraft structure damage is depicted. The method 1100 mayinclude performing a drive-around inspection with videos and images, at1102. For example, the autonomous vehicle 120 may be configured tofollow the predetermined path 202 while the camera module 110 generatesphotos and videos. Then, a determination as to whether damage issuspected may be made at 1104.

If no damage is suspected at 1104, then the method 1100 may continuewith the drive-around inspection, at 1102, until the inspection is doneat 1106. If the inspection is done, then the results may be submitted toa crew and airline for making a flight decision, at 1118. If damage issuspected, at 1104, then images may be obtained, as described herein. Aphoto image 1112 may be obtained, for example, by using the visiblelight camera 306 of FIGS. 3 and 5. A left infrared image may be taken at1108 and a right infrared image may be taken at 1110. The first infraredcamera 302 and the second infrared camera 304 may be used. Based on theimages, damage classification may be performed, at 1113. In order toclassify the damage, previously generated data (such as previous models,images, videos, etc.) may be retrieved from a database 1130 which may becloud-based. Further, grayscale enhancement may be performed at 1111 toenable comparison between the photo image taken at 1112 and data fromthe database 1130. For example, the grayscale enhancement performed at1111 may compensate for lighting under differing lighting conditions.

After the damage has been classified at 1113, an ambient backgroundremoval process may be performed at 1114. The ambient background removalprocess may remove a visible light component from the infrared imagestaken at 1108, 1110. Once the ambient background removal has taken placeat 1114, three-dimensional dimensions, such as depths and widths, of thedamage may be extracted, at 1116. An augmented reality display may begenerated at 1119 and combined with dimensions that were extracted at1116. At 1118, the crew and airline decision may be made based on theaugmented reality display. Examples of decisions that may be madeinclude a conditional minimal equipment list (MEL) departure, at 1120, aservice notification at 1124, or pushback and departure at 1122. Thislist is non-exhaustive and other decisions may be made by the crew andairline based on existing flight standards. A record may be entered intoa logbook, at 1126, noting the damage and the decision made at 1118.Further, reporting of damages and findings considered confidential toairlines may be made anonymously at 1128. These items may be stored inthe database 1130 for later use.

As shown in FIG. 11, the system 100 and its associated processes may beincorporated into existing decision-making processes used by flightcrews and ground crews to assist decision makers in their assignments.The automated process may ensure a more thorough inspection and betterdecision outcomes. Other benefits may exist.

Referring to FIG. 12, an example of a method 1200 for making aircraftservice decisions and incorporating automated assessment of aircraftstructure damage is depicted. The method 1200 may include moving anautonomous vehicle around an aircraft according to a predetermined path,where the autonomous vehicle includes a vertically extendable arm, wherea camera module is attached to a distal end of the vertically extendablearm, and where the camera module includes a first infrared camera, asecond infrared camera, and a visible light camera, at 1202. Forexample, the autonomous vehicle 120 may move around the aircraft 130according to the predetermined path 202.

The method 1200 may further include scanning an exterior surface of theaircraft using the first infrared camera, the second infrared camera,the visible light camera, or a combination thereof, at 1204. Forexample, the exterior surface 132 may be scanned using the firstinfrared camera 302, the second infrared camera 304, the visible lightcamera 306, or a combination thereof as the autonomous vehicle 120 movesaround the aircraft 130.

The method 1200 may also include determining whether a portion of theexterior surface of the aircraft is damaged based on the scan, at 1206.For example, the processor 532 may determine whether the portion 601 ofthe exterior surface 132 of the aircraft 130 is damaged based on thescan 536, which may be taken while the autonomous vehicle 120 moves.

The method 1200 may also include, in response to the portion of theexterior surface of the aircraft being damaged, using the first infraredcamera, the second infrared camera, and the visible light camera togenerate a three-dimensional model of the portion of the exteriorsurface of the aircraft, at 1208. For example, the first infrared camera302, the second infrared camera 304, and the visible light camera 306may be used to generate the three-dimensional model 542 of the damage.

The method 1200 may include additional sub-processes for generating thethree-dimensional model. For example, the method 1200 may includeinitiating movement of the autonomous vehicle to position the firstinfrared camera and the second infrared camera so that a base linebetween the first infrared camera and the second infrared camera isparallel with the portion of the exterior surface of the aircraft, at1210. For example, the autonomous vehicle 120 may be positioned suchthat the base line 610 between the first point 622 (associated with thefirst infrared camera 302) and the second point 624 (associated with thesecond infrared camera 304) is parallel with the portion 601 of theexterior surface 132 of the aircraft 130.

The method 1200 may further include rotating the first infrared cameraand the second infrared camera so that the first infrared camera and thesecond infrared camera are iteratively directed at points running alonga dent in the portion of the exterior surface of the aircraft beginningat a starting point on a first side of the dent and ending at an endingpoint on a second side of the dent, at 1212. For example, the firstinfrared camera 302 and the second infrared camera 304 may beiteratively directed at the starting point 604, the intermediate point606, and the ending point 608. Additional intermediate points may alsoexist.

The method 1200 may also include, for each point of the points runningalong the dent, determining a first angle in azimuth associated with thefirst infrared camera and a second angle in azimuth associated with thesecond infrared camera, at 1214. For example, the first angle in azimuth614 and the second angle in azimuth 618 may be determined.

The method 1200 may include calculating distances between the base lineand the points running along the dent using the first angle in azimuthand the second angle in azimuth for each point of the points, where thedistances between the base line and the points running along the dentare used to generate the three-dimensional model of the portion of theexterior surface of the aircraft, at 1216. For example, based on thefirst angle in azimuth 614 and the second angle in azimuth 618, thedistance 620 between the base line 610 and the points 604, 606, and 608may be determined. Other factors may also be included in thecalculation, such as the distance 626 between the first point 622 andthe second point 624.

The method 1200 may include determining a first distance between thefirst infrared camera and the starting point of the dent in the portionof the exterior surface of the aircraft using a first time-of-flightmeasurement, at 1218. For example, the first infrared camera 302 may usea time-of-flight measurement to determine a distance 812 between thefirst point 622 and the starting point 802.

The method 1200 may further include determining a second distancebetween the first infrared camera and the ending point of the dent usinga second time-of-flight measurement, at 1220. For example, the firstinfrared camera 302 may use a time-of-flight measurement to determine adistance 828 between the first point 622 and the ending point 804.

The method 1200 may include calculating a width between the startingpoint of the dent and the ending point of the dent based at leastpartially on the first distance and the second distance, at 1222. For anexample of FIG. 8, the distance 812 between the first point 622 and thestarting point 802 and the distance 828 between the first point 622 andthe ending point 804 may be used, among other parameters, to determinethe width 840 of the dent 602. In some cases, the other parameters mayinclude the first angle in azimuth 814. In some cases, the othercalculated parameters may include the distance 816 of either FIG. 8 or 9between the second point 624 and the ending point 804, the distance 830in FIG. 8 or the distance 930 in FIG. 9 between the second point 624 andthe starting point 802, and the distance 626 in FIG. 9 between the firstpoint 622 and the second point 624.

An advantage of the method 1200 is that by scanning camera module 110azimuthally at incremental change of the elevation angles in FIG. 4,with the help of computational algorithms and augmented reality, athree-dimensional model may be generated to enable a ground crew and/orflight crew to efficiently and effectively inspect an aircraft and makeflight decisions based on the inspection. Other advantages may exist.

Although various examples have been shown and described, the presentdisclosure is not so limited and will be understood to include all suchmodifications and variations as would be apparent to one skilled in theart.

1. An automated system comprising: a camera module comprising: a firstinfrared camera; a second infrared camera; and a visible light camera;an autonomous vehicle comprising a vertically extendable arm, whereinthe camera module is attached to a distal end of the verticallyextendable arm; and a processor configured to: initiate movement of theautonomous vehicle around an aircraft according to a predetermined path;initiate a scan of an exterior surface of the aircraft using the firstinfrared camera, the second infrared camera, the visible light camera,or a combination thereof; in response to a portion of the exteriorsurface of the aircraft being damaged, use the first infrared camera,the second infrared camera, and the visible light camera to computedimensional parameters of damage to the exterior surface and generate athree-dimensional model of the portion of the exterior surface of theaircraft.
 2. The automated system of claim 1, wherein the camera modulefurther comprises a motorized table configured to rotate in both azimuthand elevation, wherein the first infrared camera, the second infraredcamera, and the visible light camera are attached to the motorizedtable, wherein the first infrared camera and the second infrared cameraare separated by a fixed distance relative to one another, and whereininitiating the scan of the exterior surface of the aircraft comprisescommanding the camera module to scan azimuthally at incremental changesof elevation angles.
 3. The automated system of claim 1, wherein theprocessor is further configured to: plan the movement of the autonomousvehicle around the aircraft using a stored electronic map; and verifythe movement of the autonomous vehicle using detectable referencespositioned on a tarmac, a radar, a lidar, a global positioning system,or a combination thereof.
 4. The automated system of claim 1, whereinthe autonomous vehicle is configured to use proximity range sensors toavoid obstacles during the movement of the autonomous vehicle around theaircraft.
 5. The automated system of claim 1, wherein the first infraredcamera and the second infrared camera are each capable of infraredillumination using modulated continuous waves, wherein a modulatedcontinuous wave associated with the first infrared camera is phaseshifted to prevent interference with a modulated continuous waveassociated with the second infrared camera, wherein the first infraredcamera and the second infrared camera are each capable of takingindependent time-of-flight measurements using the modulated continuouswaves, and wherein the independent time-of-flight measurements are usedin generating the three-dimensional model of the portion of the exteriorsurface.
 6. The automated system of claim 1, wherein the processor isfurther configured to perform a differential measurement process usingvisible image data from the visible light camera to remove an ambientbackground light component from infrared image data received from thefirst infrared camera and the second infrared camera.
 7. The automatedsystem of claim 1, wherein, in response to the portion of the exteriorsurface of the aircraft being damaged, the processor is furtherconfigured to: initiate movement of the autonomous vehicle to positionthe first infrared camera and the second infrared camera so that a baseline between the first infrared camera and the second infrared camera isparallel with the portion of the exterior surface of the aircraft;rotate the first infrared camera and the second infrared camera so thatthe first infrared camera and the second infrared camera are iterativelydirected at points running along a dent in the portion of the exteriorsurface of the aircraft beginning at a starting point on a first side ofthe dent and ending at an ending point on a second side of the dent; foreach point of the points running along the dent, determine a first anglein azimuth associated with the first infrared camera and a second anglein azimuth associated with the second infrared camera; and calculatedistances between the base line and the points running along the dentusing the first angle in azimuth and the second angle in azimuth foreach point of the points, wherein the distances between the base lineand the points running along the dent are used to generate thethree-dimensional model of the portion of the exterior surface of theaircraft.
 8. The automated system of claim 1, wherein, in response tothe portion of the exterior surface of the aircraft being damaged, theprocessor is further configured to: determine a first distance betweenthe first infrared camera and a starting point of a dent in the portionof the exterior surface of the aircraft using a first time-of-flightmeasurement; determine a second distance between the first infraredcamera and an ending point of the dent using a second time-of-flightmeasurement; determine an angle in azimuth between a first directionassociated with the first infrared camera being directed at the startingpoint of the dent and a second direction associated with the firstinfrared camera being directed at the ending point; and calculate awidth between the starting point of the dent and the ending point of thedent based on the first distance, the second distance, and the angle inazimuth.
 9. The automated system of claim 8, wherein, in response to theportion of the exterior surface of the aircraft being damaged, theprocessor is further configured to: use the second infrared camera toconfirm the width between the starting point of the dent and the endingpoint of the dent.
 10. The automated system of claim 1, wherein, inresponse to the portion of the exterior surface of the aircraft beingdamaged, the processor is further configured to: initiate movement ofthe autonomous vehicle to position the first infrared camera and thesecond infrared camera so that a base line between the first infraredcamera and the second infrared camera is parallel with the portion ofthe exterior surface of the aircraft; determine a first distance betweenthe first infrared camera and a starting point of a dent in the portionof the exterior surface of the aircraft using a first time-of-flightmeasurement; determine a second distance between the first infraredcamera and an ending point of the dent using a second time-of-flightmeasurement; determine a third distance between the second infraredcamera and the starting point of the dent using a third time-of-flightmeasurement; determine a fourth distance between the second infraredcamera and the ending point of the dent using a fourth time-of-flightmeasurement; and calculate a width between the starting point of thedent and the ending point of the dent based on the first distance, thesecond distance, the third distance, the fourth distance, and a fifthdistance between the first infrared camera and the second infraredcamera.
 11. The automated system of claim 1, wherein the processor isfurther configured to produce an augmented reality image or video thatsuperimposes dimensional parameters from the three-dimensional modelonto an image or video of the portion of the exterior surface of theaircraft.
 12. The automated system of claim 1, wherein the processor isfurther configured to enhance visible image data generated by thevisible light camera by using a grayscale conversion to removediscrepancies due to lighting conditions.
 13. The automated system ofclaim 1, wherein the processor is further configured to provide a liveimage or video from the visible light camera to a mobile device via anetwork connection.
 14. The automated system of claim 1, wherein theprocessor is further configured to provide the three-dimensional modelto a mobile device for rendering via a network connection.
 15. Theautomated system of claim 1, wherein the processor is further configuredto provide a damage assessment to a mobile device to assist a crew, anairline, or both in making flight decisions.
 16. The automated system ofclaim 1, further comprising a database, wherein the processor is furtherconfigured to provide the three-dimensional model to the database foruse in comparisons to future three-dimensional models associated withthe portion of the exterior surface, to provide findings related todamage to the exterior surface to the database for documentation, orboth.
 17. A method comprising: moving an autonomous vehicle around anaircraft according to a predetermined path, wherein the autonomousvehicle comprises a vertically extendable arm, wherein a camera moduleis attached to a distal end of the vertically extendable arm, andwherein the camera module comprises a first infrared camera, a secondinfrared camera, and a visible light camera; scanning an exteriorsurface of the aircraft using the first infrared camera, the secondinfrared camera, the visible light camera, or a combination thereof;determining whether a portion of the exterior surface of the aircraft isdamaged based on the scan; and in response to the portion of theexterior surface of the aircraft being damaged, using the first infraredcamera, the second infrared camera, and the visible light camera togenerate a three-dimensional model of the portion of the exteriorsurface of the aircraft.
 18. The method of claim 17, further comprising:initiating movement of the autonomous vehicle to position the firstinfrared camera and the second infrared camera so that a base linebetween the first infrared camera and the second infrared camera isparallel with the portion of the exterior surface of the aircraft;rotating the first infrared camera and the second infrared camera sothat the first infrared camera and the second infrared camera areiteratively directed at points running along a dent in the portion ofthe exterior surface of the aircraft beginning at a starting point on afirst side of the dent and ending at an ending point on a second side ofthe dent; for each point of the points running along the dent,determining a first angle in azimuth associated with the first infraredcamera and a second angle in azimuth associated with the second infraredcamera; and calculating distances between the base line and the pointsrunning along the dent using the first angle in azimuth and the secondangle in azimuth for each point of the points, wherein the distancesbetween the base line and the points running along the dent are used togenerate the three-dimensional model of the portion of the exteriorsurface of the aircraft.
 19. The method of claim 17, further comprising:determining a first distance between the first infrared camera and astarting point of a dent in the portion of the exterior surface of theaircraft using a first time-of-flight measurement; determining a seconddistance between the first infrared camera and an ending point of thedent using a second time-of-flight measurement; and calculating a widthbetween the starting point of the dent and the ending point of the dentbased, at least in part, on the first distance and the second distance.20. An apparatus comprising: a camera module comprising: a firstinfrared camera; a second infrared camera; and a visible light camera;and an autonomous vehicle comprising a vertically extendable arm,wherein the camera module is attached to a distal end of the verticallyextendable arm, wherein the autonomous vehicle is configured to movearound an aircraft according to a predetermined path, and wherein thecamera module is configured to scan of an exterior surface of theaircraft using the first infrared camera, the second infrared camera,the visible light camera, or a combination thereof.